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Abstract 

The  readiness  of  a  system  under  development  cannot  be  adequately  measured  by  using 
traditional  project  management  tools  that  focus  predominantly  on  cost  and  schedule.  An 
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alternative  principally  utilized  by  NASA,  the  DoD  and  the  DoE  to  address  this  has  been  the 
prescriptive  metric  known  as  Technology  Readiness  Level  (TRL).  However,  TRL  is  only  meant 
to  measure  the  readiness  of  technology  elements  and  does  not  address  their  integration  or 
some  other  challenges  of  systems  development. 

To  address  integration,  the  Systems  Development  &  Maturity  Laboratory  (SD&ML)  at 
Stevens  Institute  of  Technology  introduced  another  prescriptive  metric  called  Integration 
Readiness  Level  (IRL).  Combining  TRL  and  IRL  scales,  SD&ML  has  formulated  a  System 
Readiness  Level  (SRL).  SRL  is  an  aggregate  measure  that  characterizes  the  progress  that  has 
been  accomplished  by  a  system  under  development  based  on  the  observable  readiness 
characteristics  of  the  technology  and  integration  elements,  not  the  cost  and  schedule  values. 

This  paper  describes  the  application  of  SRL  to  a  constrained  resource  optimization 
model  to  determine  an  optimal  development  plan  that  identifies  which  technologies  and 
integration  elements  should  be  matured  to  which  levels  such  that  a  specific  level  of  system 
readiness  is  achieved  by  a  certain  time.  This  optimal  plan  can  be  used  to  monitor  and  evaluate 
the  actual  progress  of  the  system — it  can  be  the  basis  of  a  systems  lifecycle  maturity 
management  approach  called  System  Earned  Readiness  Management  (SERM).  A  simple 
example  is  used  to  illustrate  SERM. 

1.  Introduction 

“How  much  progress  have  I  accomplished  against  my  original  plan?”  Program 
managers  ask  this  is  the  fundamental  question  in  order  to  keep  track  of  the  development  of  their 
systems.  To  answer  this,  they  have  relied  on  assessment  and  evaluation  tools.  Abba  (1997) 
describes  the  evolution  of  these  techniques  from  the  “Spend  Plan”  approach  to  Program 
Evaluation  and  Review  Technique  (PERT),  which  was  then  modified  by  the  Navy  into  PERT 
COST  in  an  attempt  to  improve  cost  management  in  1960.  Combining  its  own  experiences  with 
those  of  the  Navy’s,  the  Air  Force  in  1963  formulated  the  earliest  version  of  an  Earned  Value 
Management  (EVM)  approach  by  developing  Cost/Schedule  Planning  and  Specification 
(C/SPEC)  to  manage  the  Minuteman  program.  This  initiative  evolved  into  the  1967  Department 
of  Defense  (DoD)  Instruction  called  Cost/Schedule  Control  Systems  Criteria  or  C/SCSC  (DoD, 
1967).  Initially  developed  by  financial  managers,  C/SCSC  was  primarily  concerned  with  cost 
and  was  generally  ignored  by  project  managers  who  were  more  concerned  with  technical  and 
performance  considerations  (Abba,  1997).  In  1989,  the  organization  within  the  DoD  tasked  with 
C/SCSC  was  transferred  from  the  Controller’s  office  to  Acquisition.  By  1995,  EVM  was 
designated  as  the  preferred  tool  for  managing  risky,  cost-based  contracts  (Kaminski,  1995). 
Along  with  these  developments,  the  DoD  also  developed  the  pioneering  EVM  software 
Performance  Analyzer.  The  DoD  encouraged  the  private  sector  to  enhance  and  eventually 
replace  this  software  with  tools  that  are  commercially  available  today. 

EVM  as  a  primary  tool  has  been  credited  with  reducing  total  cost  overrun  on  the  largest, 
most  risky  DoD  contracts  to  5.5%  by  1999,  (Abba  2001).  Currently,  however,  there  is  growing 
concern  that  EVM,  which  evaluates  cost  and  schedule  performances,  does  not  adequately 
report  the  proper  maturation  of  complex  systems  under  development.  In  particular,  while  EVM  is 
quite  effective  in  capturing  and  representing  the  accomplishment  of  work  packages,  it  is  unable 
to  state  whether  these  completions  are  actually  leading  to  the  maturity  of  the  system’s  critical 
components.  Thus,  it  is  unable  to  estimate  the  maturity  or  readiness  of  the  entire  system  at  a 
given  time  during  its  development.  This  is  especially  true  when  there  is  a  high  degree  of 
uncertainty  due  to  the  novelty  and  high  technological  content  of  the  system.  Such  systems 
require  numerous  iterations  before  requirements  and  design  can  be  frozen.  Once  they  are,  then 
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EVM  becomes  a  most  effective  tool.  However,  until  that  point  in  a  system’s  development  is 
reached,  a  different  kind  of  assessment  method  is  needed. 

This  new  assessment  method  will  require  the  following  elements:  metrics  that  can 
measure  maturity  of  technologies — their  integration  links  and  the  system  itself;  the  identification 
of  optimal  development  plans  (based  on  these  metrics)  that  can  meet  the  development  strategy 
of  the  system;  and  a  mechanism  for  reporting  the  periodic  status  of  the  system  against  the 
optimal  development  plan  so  variances  can  be  measured,  explained  and  corrective  measures 
may  be  formulated. 

To  begin  to  address  these  elements  of  an  alternative  or  modified  EVM  approach,  we  will 
describe  the  application  of  a  system  maturity  metric  (i.e.,  System  Readiness  Level)  and  its 
application  to  a  constrained  resource  optimization  model  to  determine  an  optimal  development 
plan  that  identifies  which  technologies  and  integration  elements  should  be  matured  to  which 
levels,  such  that  a  specific  level  of  readiness  is  achieved  by  a  certain  time.  We  will  then  use  the 
optimal  plan  to  demonstrate  how  this  technology  can  be  used  to  monitor  and  evaluate  the  actual 
progress  of  a  system.  Thus,  it  can  become  the  basis  of  a  system’s  lifecycle  maturity 
management  approach,  which  we  have  defined  as  System  Earned  Readiness  Management 
(SERM).  We  conclude  with  a  simple  example  to  illustrate  SERM. 

2.  System  Readiness  Metrics 

In  order  to  measure  the  maturity  of  a  complex  system,  Sauser,  Verma,  Ramirez- 
Marquez,  and  Gove  (2006)  proposed  the  System  Readiness  Level  scale  or  SRL.  This  was 
eventually  refined  into  its  latest  form,  which  was  presented  to  this  Symposium  last  year  (Sauser, 
Magnaye,  Ramirez-Marquez  &  Tan,  2008b)  and  later  published  in  length  in  the  International 
Journal  of  Defense  Acquisition  Management  (Sauser,  Magnaye,  Ramirez-Marquez  &  Tan, 
2008a).  It  combines  the  widely  accepted  Technology  Readiness  Level  (TRL)  scale  (Mankins, 
1995;  2002;  DoD,  2005),  which  is  used  to  evaluate  critical  technology  elements  and  an 
Integration  Readiness  Level  (IRL)  scale  developed  by  Sauser  et  al.  (2006)  and  refined  by  Gove 
(2007).  TRL  is  presented  in  Table  1  while  IRL  is  shown  in  Table  2  below. 
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Table  1.  Technology  Readiness  Levels 


TRL 

Definition 

Description  (DoD,  2005) 

Actual  System  Proven  Through 
Successful  Mission  Operations 

Actual  application  of  the  technology  in  its  final  form  and  under  mission 
conditions,  such  as  those  encountered  in  operational  test  and 
evaluation.  In  almost  all  cases,  this  is  the  end  of  the  last  "bug  fixing" 
aspects  of  true  system  development.  Examples  include  using  the 
system  under  operational  mission  conditions. 

8 

Actual  System  Completed  and 
Qualified  Through  Test  and 
Demonstration 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of 
true  system  development.  Examples  include  developmental  test  and 
evaluation  of  the  system  in  its  intended  weapon  system  to  determine  if 
it  meets  design  specifications. 

7 

System  Prototype 

Demonstration  in  Operational 
Environment 

Prototype  near  or  at  planned  operational  system.  Represents  a  major 
step  up  from  TRL  6,  requiring  the  demonstration  of  an  actual  system 
prototype  in  an  operational  environment,  such  as  in  an  aircraft,  vehicle 
or  space.  Examples  include  testing  the  prototype  in  a  test  bed  aircraft 

6 

System/Subsystem  Model  or 
Prototype  Demonstration  in 
Relevant  Environment 

Representative  model  or  prototype  system,  which  is  well  beyond  the 
breadboard  tested  for  TRL  5,  is  tested  in  a  relevant  environment. 
Represents  a  major  step  up  in  a  technology's  demonstrated  readiness. 
Examples  include  testing  a  prototype  in  a  high  fidelity  laboratory 
environment  or  in  simulated  operational  environment. 

5 

Component  and/or  Breadboard 
Validation  in  Relevant 
Environment 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic 
supporting  elements  so  that  the  technology  can  be  tested  in  a 
simulated  environment.  Examples  include  “high  fidelity”  laboratory 
integration  of  components. 

4 

Component  and/or  Breadboard 
Validation  in  Laboratory 
Environment 

Basic  technological  components  are  integrated  to  establish  that  the 
pieces  will  work  together.  This  is  relatively  "low  fidelity"  compared  to  the 
eventual  system.  Examples  include  integration  of  “ad  hoc”  hardware  in 
a  laboratory. 

3 

Analytical  and  Experimental 
Critical  Function  and/or 
Characteristic  Proof-of-Concept 

Active  research  and  development  is  initiated.  This  includes  analytical 
studies  and  laboratory  studies  to  physically  validate  analytical 
predictions  of  separate  elements  of  the  technology.  Examples  include 
components  that  are  not  yet  integrated  or  representative. 

2 

Technology  Concept  and/or 
Application  Formulated 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  The  application  is  speculative  and  there  is 
no  proof  or  detailed  analysis  to  support  the  assumption.  Examples  are 
still  limited  to  paper  studies. 

1 

Basic  Principles  Observed  and 
Reported 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be 
translated  into  applied  research  and  development.  Examples  might 
include  paper  studies  of  a  technology's  basic  properties. 
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Table  2.  Integration  Readiness  Levels 

(Gove,  2007) 


IRL 

Definition 

Description 

9 

Integration  is  Mission  Proven  through 
successful  mission  operations. 

IRL  9  represents  the  integrated  technologies  being  used  in  the 
system  environment  successfully.  In  order  for  a  technology  to 
move  to  TRL  9  it  must  first  be  integrated  into  the  system  and 
then  proven  in  the  relevant  environment,  so  attempting  to  move 
to  IRL  9  also  implies  maturing  the  component  technology  to 

TRL  9. 

8 

Actual  integration  completed  and 
Mission  Qualified  through  test  and 
demonstration,  in  the  system 
environment. 

IRL  8  represents  not  only  the  integration  meeting  requirements, 
but  also  a  system-level  demonstration  in  the  relevant 
environment.  This  will  reveal  any  unknown  bugs/defects  that 
could  not  be  discovered  until  the  interaction  of  the  two 
integrating  technologies  was  observed  in  the  system 
environment. 

7 

The  integration  of  technologies  has 
been  Verified  and  Validated  with 
sufficient  detail  to  be  actionable. 

IRL  7  represents  a  significant  step  beyond  IRL  6;  the  integration 
has  to  work  from  a  technical  perspective,  but  also  from  a 
requirements  perspective.  IRL  7  represents  the  integration 
meeting  requirements  such  as  performance,  throughput,  and 
reliability. 

6 

The  integrating  technologies  can 

Accept,  Translate,  and  Structure 
Information  for  its  intended 
application. 

IRL  6  is  the  highest  technical  level  to  be  achieved,  it  includes 
the  ability  to  not  only  control  integration,  but  to  specify  what 
information  to  exchange,  unit  labels  to  specify  what  the 
information  is,  and  the  ability  to  translate  from  a  foreign  data 
structure  to  a  local  one. 

5 

There  is  sufficient  Control  between 
technologies  necessary  to  establish, 
manage,  and  terminate  the 
integration. 

IRL  5  simply  denotes  the  ability  of  one  or  more  of  the 
integrating  technologies  to  control  the  integration  itself;  this 
includes  establishing,  maintaining,  and  terminating. 

4 

There  is  sufficient  detail  in  the  Quality 
and  Assurance  of  the  integration 
between  technologies. 

Many  technology  integration  failures  never  progress  past  IRL  3, 
due  to  the  assumption  that  if  two  technologies  can  exchange 
information  successfully,  then  they  are  fully  integrated.  IRL  4 
goes  beyond  simple  data  exchange  and  requires  that  the  data 
sent  is  the  data  received  and  there  exists  a  mechanism  for 
checking  it. 

3 

There  is  Compatibility  (i.e.,  common 
language)  between  technologies  to 
orderly  and  efficiently  integrate  and 
interact. 

IRL  3  represents  the  minimum  required  level  to  provide 
successful  integration.  This  means  that  the  two  technologies 
are  able  to  not  only  influence  each  other,  but  also  communicate 
interpretable  data.  IRL  3  represents  the  first  tangible  step  in  the 
maturity  process. 

2 

There  is  some  level  of  specificity  to 
characterize  the  Interaction  (i.e., 
ability  to  influence)  between 
technologies  through  their  interface. 

Once  a  medium  has  been  defined,  a  “signaling”  method  must 
be  selected  such  that  two  integrating  technologies  are  able  to 
influence  each  other  over  that  medium.  Since  IRL  2  represents 
the  ability  of  two  technologies  to  influence  each  other  over  a 
given  medium,  this  represents  integration  proof-of-concept. 

1 

An  Interface  between  technologies 
has  been  identified  with  sufficient 
detail  to  allow  characterization  of  the 
relationship. 

This  is  the  lowest  level  of  integration  readiness  and  describes 
the  selection  of  a  medium  for  integration. 

The  SRL  scale  is  calculated  by  using  a  normalized  matrix  of  pair-wise  comparisons  of 
TRLs  and  IRLs  that  reflects  the  actual  architecture  of  the  system.  Briefly  stated,  the  IRL  matrix 
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is  obtained  as  a  symmetric  square  matrix  (of  size  n*n)  of  all  possible  integrations  between  any 
two  technologies  in  the  system.  For  technology  integration  to  itself,  perfect  integration  is 
assumed  (IRL=  9)  while  an  IRL  of  zero  is  used  when  there  is  no  integration  between  two 
technologies.  Likewise,  the  vector  TRL  defines  the  readiness  level  of  each  of  the  technologies  in 
the  system.  In  its  current  form,  the  SRL  is  calculated  as 


[srl] 


'SRL,' 

' IRL nTRL,  +IRLl2TRL,  + ...  + IRLlnTRLn  ' 

SRL, 

= 

IRL21TRL,  +  IRL22TRL2  + ...  +  IRL2nTRLn 

_SRLn_ 

IRLnlTRL,  +  IRL  n2TRL2  + ...  +  IRLnnTRLn  _ 

where  IRLij=IRLji 


and 


SRL,  SRL,  SRL, ) 

- L  + - ?-  +  ...  + - n- 


n 

where  n,is  the  number  of  integrations  with  technology  /  plus  its  integration  to 

itself. 

The  resulting  SRL  metric  can  be  used  to  determine  the  maturity  of  a  system  and  its 
status  within  the  developmental  lifecycle.  Table  3,  for  example,  is  a  representation  of  how  the 
SRL  scale  correlates  to  a  systems  engineering  lifecycle.  These  notional  values  of  the  SRL  scale 
shown  in  Table  3  are  meant  to  be  organization-generic  examples  of  how  the  calculated  SRL 
values  can  be  set  as  a  guide  by  a  systems  engineer  or  program  manager.  That  is,  in  practice 
the  systems  engineer  or  program  manager  at  the  outset  must  determine  what  values  of  the  SRL 
correlate  to  that  point  where  one  phase  begins  and  where  it  ends  for  that  particular  system.  A 
calibration  of  these  relevant  ranges  for  each  phase  of  system  development  will  have  to  be 
program-specific  or,  at  best,  pertinent  only  to  a  particular  class  of  systems  that  share  a  large 
degree  of  similarity.  Therefore,  the  SRL  value  of  a  system  can  only  be  compared  to  that  of  the 
same  system  or  a  very  similar  system. 
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Table  3.  System  Readiness  Levels 


SRL 

Name 

Definitions 

0.90  to  1.00 

Operations  &  Support 

Execute  a  support  program  that  meets  materiel  readiness 
and  operational  support  performance  requirements  and 
sustains  the  system  in  the  most  cost-effective  manner  over 
its  total  lifecycle. 

0.80  to  0.89 

Production  &  Deployment 

Achieve  operational  capability  that  satisfies  mission  needs. 

0.60  to  0.79 

Engineering  & 
Manufacturing 
Development 

Develop  system  capability  or  (increments  thereof);  reduce 
integration  and  manufacturing  risk;  ensure  operational 
supportability;  minimize  logistics  footprint;  implement  human 
systems  integration;  design  for  production;  ensure 
affordability  and  protection  of  critical  program  information; 
and  demonstrate  system  integration,  interoperability,  safety 
and  utility. 

0.40  to  0.59 

Technology  Development 

Reduce  technology  risks  and  determine  and  mature 
appropriate  set  of  technologies  to  integrate  into  a  full  system 
and  demonstrate  CTEs  on  prototypes. 

0.10  to  0.39 

Materiel  Solution  Analysis 

Assess  potential  materiel  solution  options 

NOTE:  These  ranges  have  been  derived  conceptually  and  are  undergoing  field  verification  and 


validation  under  Naval  Postgraduate  School  Contract  #  N00244-08-0005. 

While  the  TRL  has  been  widely  accepted  by  many  government  and  industry 
organizations,  the  IRL  and  SRL  need  continued  verification  and  validation;  efforts  are  currently 
under  way.  Early  results  indicate  that  SRL  can  institute  a  robust  and  repeatable  method  for 
assessment  and  reporting  the  status  of  a  system’s  development.  It  enables  program  managers 
to  evaluate  system  development  in  real  time  and  take  corrective  actions.  It  can  also  be  applied 
as  a  predictive  tool  for  technology  insertion  (Michaud,  Forbes,  Sauser  &  Gentile,  2008).  In  order 
to  firmly  establish  the  validity  of  SRL,  it  must  be  applied  to  a  sufficient  number  of  real  complex 
systems  under  development. 

Nevertheless,  a  rudimentary  SRL  calculator  has  been  developed  by  the  Systems 
Development  &  Maturity  Laboratory  (SD&ML)  at  Stevens  Institute  of  Technology  (see 
http://www.SystemReadinessLevel.com;  Tools)  and  is  undergoing  refinement.  In  addition,  the 
SD&ML  is  in  ongoing  partnerships  to  develop  tools  for  system  maturity  assessment  that 
leverage  their  continued  research  in  systems  maturity. 


3.  Formulating  Optimal  Development  Plans 

System  development  is  pursued  based  on  two  generic  strategies:  minimizing  costs  or 
being  the  first  to  market/deployment  (Laugen,  Acur,  Boer  &  Fick,  2005).  In  order  to  meet  these 
strategic  imperatives,  the  program  manager  must  have  the  capability  to  instruct  the  project 
managers  about  which  technologies  and  integration  links  must  be  matured  to  sufficient  levels 
and  when.  Leveraging  the  SRL  method  previously  described,  such  a  development  plan  can  be 
formulated  by  relying  on  constrained  optimization  techniques.  The  methodology  for  cost 
minimization  has  been  formulated  by  Magnaye,  Sauser,  and  Ramirez-Marquez  (2009)  while  the 
first  to  market/deployment  was  developed  by  Sauser  and  Ramirez-Marquez  (2009).  These  are 
summarized  below. 
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3.1.  Cost-driven  Strategy 

The  cost-driven  strategy  is  becoming  more  common  as  political  pressure  (in  government 
programs)  and  competitive  intensity  (in  industry)  becomes  more  pronounced  in  the  current  and 
future  economic  environment  characterized  by  more  constrained  resources  and  more 
demanding  customers.  In  this  case,  the  development  strategy  is  to  optimize  the  allocation  of 
limited  resources  while  attaining  a  certain  level  of  system  maturity  or  readiness  within  a 
specified  time.  In  order  to  execute  the  development  required  to  reach  a  SRL  value  by  a  certain 
time,  it  is  necessary  to  know  how  to  reach  this  level  at  a  minimum  cost.  To  address  these 
concerns,  Magnaye  et  al.,  (2009)  proposed  an  optimization  model  whose  objective  is  to 
minimize  development  cost  (a  function  of  TRL  and  IRL  development)  under  constraints 
associated  with  the  required  SRL  value  and  schedule.  This  model  recognizes  that  technologies 
compete  for  resources  and  that  the  optimal  allocation  of  the  least  amount  of  resources  to  reach 
a  certain  SRL  value  is  desirable.  The  general  mathematical  form  of  this  model  called  SCODmin 
follows: 


Minimize:  SCOD  (TRL, IRL)  =  SCODfixed  +  SCODvariable  (TRL, IRL) 
Subject  to:  SRL(TRL,IRL)  >  A 
R-,  (TRL, IRL)  <  r, 


Rh  (TRL, IRL)  <  rh 

In  addition  to  the  SRL  and  time  or  schedule  constraints,  other  possible  constraints  could 
be  technical  performance  parameters  such  as  equivalent  mass  for  space  systems,  peak  load 
capacities  for  transportation  and  so  on. 


The  matrices  IRL  and  TRL  in  Model  SCODmin  contain  the  decision  variables.  Each 
variable  is  integer-valued  and  bounded  by  (/RL„ 9)  and  (TRL/, 9),  respectively.  That  is,  the 
TRL/IRL  for  the  /th  component  cannot  be  below  its  current  level  or  above  perfect  technology  or 
integration  development  (IRL  or  TRL  =  9). 


To  completely  characterize  the  decision  variables  in  Model  SCODmin,  it  is  necessary  to 
introduce  the  following  transformation: 


If  TRLi  =  k 
otherwise 


and  x;=‘ 


U  IRL  y  =  k 
otherwise 


for  /c=  1 , ...  9 


Notice  that  based  on  these  binary  variables,  each  of  the  possible  normalized  TRL  and 


IRL  in  the  system  can  be  obtained  as  TRLt  = 
variables  SRL,  is  transformed  to: 


k=l 


9 


and  IRL0  =— - .  Based  on  these  binary 

9 
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Based  on  the  computation  of  the  SRL  with  these  decision  variables,  Model  SCODmin 
belongs  to  the  class  of  binary,  integer-valued,  non-linear  problems.  For  a  system  with  n 
technologies  containing  m  (m<(n-1)n/2)  distinct  integrations,  and  assuming  all  technologies  and 
integrations  are  at  their  lowest  levels,  there  are  9n+m  potential  solutions  to  Model  SCODmin. 
Evaluating  each  possible  solution  is  prohibitive,  so  to  generate  a  more  timely  optimal  solution,  a 
meta-heuristic  approach  developed  by  Ramirez-Marquez  and  Rocco  (2008)  was  applied  to  the 
system  under  development  and  is  described  below.  This  approach,  called  Probabilistic  Solution 
Discovery  Algorithm  (PSDA)  has  the  capability  of  producing  quasi-optimal  solutions  in  a 
relatively  short  period  of  time.  However,  it  must  be  mentioned  that  the  results  cannot  be  proven 
as  the  optimal  solution  because  by  taking  a  probabilistic  approach,  the  algorithm  can  only  select 
subsets  of  the  entire  feasible  set  from  which  to  find  a  solution.  Every  time  the  algorithm  is  run  a 
different  subset  is  selected.  Nevertheless,  prior  tests  have  indicated  that  PSDA  results  tend  to 
be  better  than  results  from  alternative  meta-heuristic  approaches  (Ramirez-Marquez  &  Rocco, 
2007). 


As  used  in  the  solution  of  the  minimization  problem,  the  algorithm  follows  three  inter¬ 
related  steps: 

■  Strategy  Development — a  Monte  Carlo  simulation  is  used  to  identify  the 
potential  TRL  or  IRL  levels  the  technologies  and  links  can  advance  or  mature; 

■  Analysis — each  potential  solution  is  analyzed  by  calculating  its  associated 
cost,  schedule  and  SRL; 

■  Selection — through  an  evolutionary  optimization  technique,  a  new  optimal  set 
of  technologies  and  integration  links  (with  their  corresponding  TRLs  and  IRLs 
are  chosen  based  on  the  cost,  schedule  and  SRL  values). 


DEFENSE  ACQUISITION  IN  TRANSITION 


- 103- 


3.2.  Notional  Example  and  Results 

Figure  1.  System  Concept  Diagram 


Tech  1:  Remote  Manipulator  System  (RMS);  Tech  2:  Special  Purpose  Dexterous  Manipulator  (SPDM); 
Tech  3:  Electronic  Control  Unit  (ECU);  Tech  4:  Autonomous  Grappling  (AG);  Tech  5:  Autonomous 
Proximity  Operations  (APO);  and  Tech  6:  Laser  Image  Detection  and  Radar  (LIDAR). 


The  following  notional  example  will  use  a  simple  system  of  six  technologies  and  seven 
integrations  (see  Figure  1  above)  to  demonstrate  the  steps  involved  in  calculating  the  SRL  value 
and  minimizing  the  cost  subject  to  constraints  on  system  maturity  and  schedule.  By  evaluating 
the  SRL  of  this  system,  an  estimate  of  its  actual  readiness  can  be  obtained  before  being 
deployed.  In  year  1  (current  year),  when  reviewing  the  SRL  for  this  system  in  its  current  state, 
the  calculations  yielded  an  SRL  of  0.48.  Referring  to  Table  3,  this  value  indicates  that  this 
system  should  be  in  the  Technology  Development  phase,  with  the  technologies  close  to 
maturity  (lowest  TRL  is  6)  while  integration  elements  are  behind,  one  as  low  as  level  2  only.  For 
the  system  used  in  this  example,  Tables  4  and  5  present  the  incremental  budgetary  and  time 
requirements  to  mature  each  technology  and  integration  element  from  its  current  level  to  the 
next.  For  example,  to  mature  Technology  1  from  its  current  TRL  of  8  to  9  will  require  another 
$900,000  and  349  labor-hours.  In  order  to  fully  mature  all  the  technologies  and  integration 
elements,  an  additional  $26,574  million  and  19,122  labor-hours  are  required. 
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Table  4.  Estimated  Incremental  Cost  (xIOOO)  and  Time 
for  Each  Technology  Effort 


Technology 

1 

2 

3 

4 

5 

6 

TRL  Level 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

1 

2 

3 

4 

5 

6 

7 

$876 

127 

$467 

280 

$780 

450 

8 

$689 

476 

$421 

341 

$531 

236 

$123 

21 

9 

$900 

349 

$765 

432 

$734 

299 

$853 

568 

$189 

48 

$389 

300 

Table  5.  Estimated  Incremental  Cost  (xIOOO)  and  Time 
for  Each  Integration  Effort 


Integration 

1,2 

1,3 

2,3 

2,4 

3,5 

4,5 

5,6 

IRL  Level 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

1 

2 

3 

$453 

200 

$123 

80 

4 

$581 

400 

$219 

380 

5 

$721 

658 

$595 

532 

6 

$100 

140 

$275 

164 

$900 

700 

$700 

621 

7 

$175 

180 

$200 

93 

$50 

25 

$540 

320 

$345 

324 

$1,200 

954 

$808 

862 

8 

$400 

300 

$400 

165 

$450 

320 

$632 

432 

$457 

400 

$1 ,432 

1021 

$1,003 

997 

9 

$600 

500 

$650 

389 

$550 

465 

$745 

690 

$678 

500 

$1,765 

1238 

$1,110 

1145 

If,  for  example,  management  wants  to  increase  maturity  from  the  current  value  of  0.48 
(Technology  Development  stage)  to  0.69  (Engineering  &  Manufacturing  Development  stage), 
using  a  maximum  of  40%  of  the  remaining  time  (7,649  labor-hours),  the  PSDA  cost  minimization 
model  calculated  a  minimum  additional  development  cost  of  $5,914  million  and  would  require 
3,797  labor-hours. 

In  addition,  the  development  plan  that  can  achieve  this  desired  SRL  value  of  0.69  with 
the  least  cost  will  be  attained  if  the  subsystems  that  are  based  on  each  technology  element 
reach  the  maturity  levels  listed  in  Table  6.  The  latter  shows  that  of  the  six  subsystems,  two  are 
ahead  (SRL13j),  two  are  behind  (SRL4j5)  and  two  are  close  to  the  same  level  (SRL2,6)  as  the 
whole  system.  This  insight  can  become  useful  when  the  maturity  levels  are  associated  with 
systems  engineering  activities.  That  is,  the  spectrum  of  SRLi’s  can  indicate  levels  of  variation  in 
the  systems  engineering  activities,  which  are  needed  to  mature  the  entire  system. 
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Table  6.  Subsystem  and  Composite  SRLs 


SRL1 

SRL2 

SRL3 

SRL4 

SRL5 

SRL6 

I 

Composite 
SRL  =  1/6 

0.856 

0.707 

0.815 

0.461 

0.593 

0.722 

4.154 

0.692 

Table  7  summarizes  the  additional  results  for  the  targeted  SRL  values  and  Table  9 
indicates  the  development  plan  for  each  improvement  scenario. 


Table  7.  Best  Solutions  for  Desired  SRL  Values 


Year 

SRL 

Time  (man-hrs) 

Computed  Minimum 
Cost  ($  xIOOO) 

Targeted 

Computed 

Targeted 

Computed 

1 

0.48 

0.48 

NA 

NA 

NA 

2 

0.58 

0.587 

3,824 

1,654 

2,203 

3 

0.69 

0.692 

7,649 

3,797 

5,914 

4 

0.79 

0.794 

11,473 

7,667 

11,065 

5 

0.89 

0.896 

15,298 

11,309 

16,888 

6 

1.00 

1.00 

19,122 

19,122 

26,574 

Table  8.  Development  Plan 


Year 

Target 

TRL 

IRL 

SRL 

1 

2 

3 

4 

5 

6 

1,2 

1,3 

2,3 

2,4 

3,5 

4,5 

5,6 

6 

1.000 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

5 

0.89 

9 

9 

9 

8 

9 

9 

9 

9 

9 

8 

8 

5 

7 

4 

0.79 

8 

9 

9 

6 

9 

9 

9 

9 

9 

5 

8 

4 

6 

3 

0.69 

8 

8 

9 

6 

9 

9 

8 

8 

7 

5 

7 

2 

4 

2 

0.58 

8 

8 

8 

6 

7 

6 

7 

7 

7 

5 

6 

2 

4 

1 

0.48 

8 

8 

7 

6 

6 

6 

5 

6 

6 

5 

6 

2 

2 

It  must  be  noted  that  the  algorithm  can  only  work  if  the  management  objectives  are 
inherently  feasible.  If  a  prescribed  objective  is  impossible  to  achieve — as  when  too  little  time  or 
labor-hours  are  available — the  algorithm  will  not  produce  a  solution. 


3.3  First-to-Market/Deployment  Strategy 

A  very  similar  optimization  procedure  can  be  designed  to  determine  how  fast  a  system 
can  reach  a  certain  stage  in  the  development  lifecycle  or  how  quickly  it  can  be  deployed.  In  this 
case,  there  may  be  a  need  to  launch  an  experimental  system  in  favor  of  maximum  current  and 
short-term  effectiveness  while  disregarding  long-term  reliability.  The  objective  may  be  to  meet 
pressing  needs  in  a  war  theater  or  commercial  market  as  quickly  as  possible. 

For  example,  there  is  currently  a  necessity  to  deliver  Operationally  Responsive  Space 
(ORS)  systems  to  meet  shortfalls  in  tactical  space  capabilities  (e.g.,  communications  and 
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imagery)  that  the  warfighter  needs  in  Iraq  and  Afghanistan.  These  are  being  satisfied  through 
the  development  of  small  experimental  satellites  called  TacSats  (average  cost=$87  million)  as 
well  as  improvements  in  the  capabilities  of  small  launch  vehicles  (GAO,  2008).  In  the  private 
sector,  the  first  company  to  develop  commercially  viable  autonomous-recharging  powertrain 
battery  systems  will  enjoy  first-mover  advantages  in  the  defense  and  commercial  motor  vehicle 
industry.  Such  a  company  will  be  able  to  create  and  sustain  barriers  to  entry  through  control  of 
the  technology  (property  rights),  brand  recognition  and  so  on. 

In  such  instances,  the  primary  objective  is  to  maximize  the  readiness  of  the  system 
utilizing  a  given  amount  of  limited  resources.  Sauser  and  Ramirez-Marquez  (2009)  developed 
an  SRL  maximization  model — SRLmax — for  such  an  application.  As  with  the  SCODmin  model,  this 
model  recognizes  that  the  technologies  as  well  as  the  integration  elements  that  form  the  system 
compete  for  resources  and  that  in  order  to  reach  the  highest  level  of  readiness,  a  program 
manager  must  be  able  to  allocate  the  limited  resources  optimally.  Just  as  what  had  to  be  done 
in  the  SCODmin  model,  the  program  manager  must  be  able  to  decide  which  technologies  and 
integrations  can  be  advanced  to  which  levels  of  readiness  at  a  certain  point  in  time  in  order  to 
reach  the  highest  level  of  readiness  for  the  system.  The  general  mathematical  form  of  SRLmax 
follows: 


Maximize:  SRL  (TRL.IRL) 
Subject  to:  Ri  (TRL,IRL)  <  n 


Rh  (TRL.IRL)  <  rh 

As  with  the  minimization  model  above,  SRLmax  belongs  to  the  class  of  integer-valued, 
non-linear  problems. 

Using  the  same  data  for  the  notional  example  above,  the  maximization  algorithm 
indicated  that  to  get  to  the  Engineering  &  Manufacturing  Development  phase  of  the  lifecycle  with 
an  SRL  value  of  0.73,  $7,724  million  and  5,081  labor-hours  will  be  required.  For  comparison 
purposes,  the  optimal  development  plans  to  get  to  the  Engineering  &  Manufacturing 
Development  stage,  albeit  at  different  SRL  values  (0.69  for  cost  minimization  and  0.73  for  SRL 
maximization)  are  presented  in  Table  9  below. 


Table  9.  Comparable  Development  Plans 


Model 

SRL 

TRL 

IRL 

1 

2 

3 

4 

5 

6 

1,2 

1,3 

2,3 

2,4 

3,5 

4$ 

5,6 

SCODmjn 

0.69 

8 

8 

9 

6 

9 

9 

8 

8 

7 

5 

7 

SRLmax 

0.73 

8 

9 

9 

6 

9 

9 

8 

8 

8 

5 

7 

5 

The  cost  minimization  strategy  will  reach  this  stage  by  year  3.  On  the  other  hand,  when 
the  objective  is  to  deploy  as  quickly  as  possible,  the  system  can  be  in  production  as  soon  as  the 
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prescribed  resources  are  applied,  provided  the  process  and  product  technologies  are  amenable 
to  accelerating  the  schedule.  This  assumes  constant  productivity  that  represents  an  ideal 
situation.  In  reality,  there  is  more  likely  to  be  “process  congestion,”  which  can  lead  to  increased 
coordination  and  communication  expenses,  among  other  things.  Therefore,  for  the  maximization 
model,  the  estimated  incremental  costs  for  each  TRL  and  IRL  level  must  be  adjusted  upwards  in 
order  to  reflect  the  cost  implications  of  “crashing”  the  schedule.  Depending  upon  these  CTE- 
and  integration  link-specific  cost  increases,  the  formulated  development  plan  is  likely  to  be 
different  from  the  one  obtained  from  the  minimization  model. 

The  results  must  be  carefully  examined  by  the  program  manager  and  adjusted  according 
to  a  proper  understanding  of  the  technologies  involved  and  the  context  for  the  system.  For 
example,  in  the  previous  illustration,  some  of  the  integration  links  have  to  be  examined  more 
closely  and  compared  to  a  pre-determined  minimum  acceptable  readiness  values.  If  the 
minimum  IRL  values  of,  say,  5  are  required  in  order  to  proceed  to  production  within  acceptable 
risk  limits,  then,  additional  resources  must  be  allocated  to  mature  integration  links  (4,5)  and 
(5,6)  to  this  level.  This  threshold  IRL  value  may  be  higher  for  a  cost  minimization  strategy 
(whereas  long-term  reliability  is  an  important  lifecycle  variable)  and  lower  for  the  first-to- 
deployment  experimental  strategy  that  characterizes  the  TacSats  program  in  which  long-term 
reliability  is  not  quite  as  important  as  delivering  the  capability  sooner  rather  than  later. 

It  must  also  be  noted  that  the  solution  is  driven  by  the  estimates  of  cost  and  labor  inputs. 
The  effectiveness  of  the  optimization  models  are  very  dependent  on  the  accuracy  of  the 
estimates  of  the  resources  required  to  proceed  from  one  readiness  level  to  the  next.  If  these 
values  are  unrealistic,  sub-optimal  solutions  will  be  generated. 

Furthermore,  given  the  high  levels  of  uncertainty  associated  with  complex  systems  that 
are  under  development,  estimates  of  costs  which  are  farther  into  the  future  may  be  less  reliable 
than  those  which  are  closer  to  the  current  period.  Thus,  estimates  have  to  be  continually  refined 
and  reapplied  to  the  optimization  algorithm  in  order  to  fine-tune  the  development  plan 
accordingly. 

4.  Monitoring  Progress 

The  metrics  that  measure  readiness  together  with  the  development  plans  generated  by 
the  appropriate  optimization  model  serve  as  the  foundation  for  a  mechanism  that  can  measure 
and  communicate  accomplishments  during  the  development  of  complex  systems.  As  a  general 
principle,  EVM  may  be  retained  as  the  preferred  tool  for  project  managers  tasked  with 
developing  each  of  the  critical  technology  and  integration  elements.  To  consider  all  the  projects 
that  an  enterprise  has  to  manage,  Project  Portfolio  Management  (PPM)  has  been  suggested  by 
De  Reyck  et  al.  (2005)  and  Martinsuo  and  Lehtonen  (2007).  In  between,  to  manage  the 
development  of  a  system,  which  is  a  set  of  projects  that  are  related  because  they  share  a 
common  objective  or  client — a  program  management  tool  is  required.  Developing  such  a  tool, 
which  we  refer  to  as  System  Earned  Readiness  Management  (SERM),  is  one  of  the  activities 
we  intend  to  pursue  next.  SERM  is  intended  to  be  very  similar  to  EVM.  It  must  answer  the 
following  questions: 

■  What  amount  of  readiness  is  expected  from  the  tasks  planned? 

■  What  level  of  readiness  was  accomplished  by  the  tasks  completed? 

■  How  many  resources  did  the  accomplished  level  of  readiness  cost? 
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■  How  many  resources  were  allocated  to  reach  this  level  of  readiness? 

■  What  was  the  total  budgeted  resources  to  fully  mature  the  system? 

■  What  are  now  the  expected  total  resources  required  to  develop  the  system? 

4.1  Work  Breakdown  Structure  for  SERM 

SERM  will  require  a  breakdown  of  the  tasks  necessary  to  define  the  system,  develop  the 
critical  technology  elements  and  integrate  them  into  the  desired  system.  The  tasks  could  be 
oriented  towards  the  phases  of  the  system  lifecycle  at  the  highest  levels  (i.e.,  Materiel  Solution 
Analysis,  Technology  Development,  Engineering  &  Manufacturing  Development,  Production  & 
Deployment,  and  Operations/Support)  and  continue  to  be  disaggregated  into  the  TRL  and  IRL 
levels  that  have  to  be  attained  and,  if  necessary,  down  to  the  jobs  that  must  be  completed  to 
reach  the  desired  readiness  for  each  time  period.  An  abbreviated  example  is  shown  in  Table  10 
below. 
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Table  10.  WBS  for  SERM 


1 . SYSTEM  A 

1.1  Materiel  Solution  Analysis  Phase 

1.1.1  Materiel  Solution  Analysis  Decision  Review 

1 .1 .1 .1  Joint  Requirements  Oversight  Council  (JROC)  recommendations 

1.1. 1.2  Initial  Capabilities  Document(ICD) 

1 .1 .1 .2.1  Preliminary  concept  of  operations 

1 .1 .1 .2.2  Description  of  needed  capability 

1 .1 .1 .3  Analysis  of  Alternatives  (AoA) 

1 .1 .1 .3.1  Determine  acquisition  phase  of  entry 

1 .1 .1 .3.2  Identify  the  initial  review  milestone 

1 .1 .1 .3.3  Designate  the  lead  DoD  Component(s) 

1.1. 1.3.4  Prepare  Acquisition  Decision  Memorandum 

1.1.2  Satisfy  phase-specific  entrance  criteria  for  initial  review  milestone 

1.1. 2.1  Proposed  materiel  solution 

1 .1 .2.2  Secure  full  funding  for  Technology  Development  Phase 


1.2.  Technology  Development  Phase 

1.2.1  Management 

1.2.1  Materiel  solution 

1.2.2Technology/system  development  strategy 
1.2.3  Acquisition  decision  memorandum 

1.2.2  CTE  1 

1. 2.2.1  TRL  =3 

1. 2.2.2  TRL  =4 

1. 2.2.3  TRL  =5 

1.2.3  CTE  2 

1. 2.3.1  TRL  =3 

1. 2.3.2  TRL  =4 

1. 2.3.3  TRL  =5 

1. 2.3.4  TRL  =6 

1.2.x  CTE  n . etc. 


1.3.  Engineering  &  Manufacturing  Phase 

1.3.1  Management 

1.3. 1.1  Key  performance  parameters  ...  etc. 

1.3.2  CTE  1 

1. 3.2.1  TRL  =  6  ...etc 


1.4.  Production  &  Deployment  Phase 


1.5.  Operations  &  Support  Phase 


4.2  Determining  Earned  Readiness  and  Baseline 

A  readiness-oriented  baseline  should  reflect  the  cumulative  increase  in  the  readiness  of 
the  technology  and  integration  elements  of  the  system.  Readiness  is  allocated  throughout 
the  system  by  assigning  the  TRL  and  IRL  values  to  the  tasks  completed  if  and  only  if  they 
satisfy  the  definition  for  that  readiness  level.  Thus,  it  is  possible  that  under  SERM,  a  planned 
task  may  be  completed  during  the  specified  time  frame  but  if  it  fails  to  advance  the  maturity  of 
that  particular  technology  or  integration  link,  that  completed  task  did  not  earn  any  readiness 
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values.  By  doing  this,  a  program  manager  can  clearly  see  which  activities  have  failed,  in  order 
to  identify  the  sources  of  cost  overruns  and  find  and  communicate  explanations  for  exceeding 
the  budget. 

This  scenario  is  not  unlikely  given  the  high  amount  of  uncertainty  involved  with 
developing  complex  systems.  This  uncertainty — the  result  of  novelty,  high  technological  content 
and  very  long  development  lifecycles — can  lead  to  late  identification  of  requirements  and  design 
flaws,  requirements  churn  (due  to  inaccurate  statements  of  user  needs),  delays  in  integration 
and  testing  and  the  need  for  significant  unplanned  work — rework  as  well  as  revisions  in  the 
system  architecture  and  technology  choices  (Brownsword  &  Smith,  2005). 

5.  Conclusion 

This  paper  suggested  the  development  of  a  new  program  assessment  and  evaluation 
system  that  relies  on  the  readiness  measurement  of  a  system’s  critical  technology  elements 
(using  TRL)  and  the  integrations  that  link  them  to  each  other  (using  IRL),  which  are  then 
combined  to  estimate  a  System  Readiness  Level  (SRL)  in  order  to  determine  the  readiness  of 
the  system  as  a  whole.  SRL  can  then  be  combined  with  the  prescribed  strategy  for  developing 
the  system  (either  minimize  costs  or  be  the  first  to  deploy  the  system)  and  used  in  an 
appropriate  constrained  optimization  model  to  formulate  the  optimal  development  plan.  Based 
on  this  plan,  the  progress  of  the  system  development  effort  can  be  monitored  and  evaluated 
using  System  Earned  Readiness  Management  (SERM). 

Of  the  various  concepts  enumerated  here,  only  TRL  has  been  accepted  as  a  generally 
valid  principle.  IRL,  SRL  and  SERM  are  all  new,  and  thus  require  substantial  efforts  to  verify  and 
validate.  It  is  necessary  to  apply  them  to  a  sufficient  number  of  programs  that  have  recently 
been  completed  or  are  currently  being  implemented.  It  must  be  noted  that  EVM  for  project 
management  became  more  widely  accepted  only  when  the  graduate  students  in  the  DoD’s 
academic  institutions  were  able  to  apply  it  to  defense  acquisition  projects  and  show  its  benefits 
(Abba,  2001 ).  The  early  anecdotal  evidence  from  the  few  attempts  to  apply  SRL  has  been 
positive  and  may  justify  a  similar  approach  to  verify  and  validate  it. 
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TRL  Shortcomings 


•  Application  of  TRL  to  systems  of  technologies  is  not 
sufficient  to  give  a  holistic  picture  of  complex  system 
of  systems  readiness 

°  TRL  is  only  a  measure  of  an  individual  technology 

•  Assessments  of  several  technologies  rapidly 
becomes  very  complex  without  a  systematic  method 
of  comparison 

•  Multiple  TRLs  do  not  provide  insight  into 
integrations  between  technologies  nor  the  maturity 
of  the  resulting  system 

°  Yet  most  complex  systems  fail  at  the  integration  points 
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But,  what's  missing... 


•  “Readiness”  values  tend  to  be  soft  metrics1  that 
are: 

n  Relatively  easy  to  derive,  but  require  a 
complementing  rational  that  explains  the 
assessment, 

°  Human-intensive, 
n  Subjective, 

°  Contain  inherent  variations  or  ambiguity  that  is 
averaged  away. 


fowling,  T.  &  Pardoe,  T.  (2005)  'TIMPA  -  Technology  Insertion  Metrics  Volume  1',  Ministry  of  Defence,  United  Kingdom, 
OinetiO. 
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Integration  Readiness  Level 


IRL 

Definition 

9 

Integration  is  Mission  Proven  through  successful  mission  operations. 

8 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration, 
in  the  system  environment. 

7 

The  integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail 
to  be  actionable. 

6 

The  integrating  technologies  can  Accept,  Translate,  and  Structure  Information  for  its 
intended  application. 

5 

There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and 
terminate  the  integration. 

4 

There  is  sufficient  detail  in  the  Quality  and  Assurance  of  the  integration  between 
technologies. 

3 

There  is  Compatibility  (i.e.  common  language)  between  technologies  to  orderly  and 
efficiently  integrate  and  interact. 

2 

There  is  some  level  of  specificity  to  characterize  the  Interaction  (i.e.  ability  to 
influence)  between  technologies  through  their  interface. 

1 

An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow 
characterization  of  the  relationship. 

Gove,  R.  (2007)  Development  of  an  Integration  Ontology  for  Systems  Operational  Effectiveness.  M.S.  Thesis.  Stevens  Institute  of  Technology.  Hoboken,  NJ 
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What  we  are  doing? 

Development  of  metrics,  tool,  and  methodologies 
for  determining  a  systems  readiness  level  (SRL)  and 
potential  for  making  efficient  and  effective  life- 
cycle  acquisition  and  operational  decisions.  The  SRL 
Model  is  a  function  of  the  individual  Technology 
Readiness  Levels  (TRL)  and  their  subsequent 
integration  points  with  other  technologies,  the 
Integration  Readiness  Level  (IRL). 


Integration 
Readiness 
Level  (IRL) 

Value  Proposition: 

■  Currently  TRL  is  only  a  measure  of  an  individual  technology 

■  There  is  no  method  for  integrating  TRLs 

■  There  is  no  systematic  measure  of  a  systems  readiness 

■  Cost  and  schedule  reduction  in  strategic  technology 
development  planning 

Deliverable:  Integration  of  methodologies  for  strategic 
roadmap  planning  that  illustrate  the  timely 
implementation  of  capability  increments. 


Technology^ 
Readiness 
Level  (TRL) 


The  System 


t 

SRL  =  f  (TRL,  IRL) 
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SRL  Calculation 

The  SRL  is  not  user  defined,  but  is  instead  based  on  the  outcomes  of  the 
documented  TRL  and  IRL  evaluations 

Through  mathematically  combining  these  two  separate  readiness  levels,  a 
better  picture  of  overall  complex  system  readiness  is  obtained  by 
examining  all  technologies  in  concert  with  all  of  their  required  integrations 

SRL  =  IRL  x  TRL 


r 

SRL, 

srl2 

SRL3 

_ 

r 

IRL11 

IRL12 

IRL12 

IRL22 

"\ 

irl13 

irl23 

X 

r 

IRL, 

trl2 

k. 

J 

IRLi3 

k.  3 

irl23 

IRL3^ 

TRL 

C  J 

Composite  SRL  =  1/n  |  SRL^n  +  SRL^/n  +  SRL^/n 
=  1/n2 1  SRLj  +  SRLj  +  SRL^ 

These  values  serve  as  a  decision-making  tool  as  they  provide  a 
prioritization  guide  of  the  system’s  technologies  and  integrations  and  point 
out  deficiencies  in  the  maturation  process 


Sauser,  B.,  J.  Ramirez-Marquez,  R.  Magnaye,  and  W.  Tan.  (2008).  "A  Systems  Approach  to  Expanding  the 
Technology  Readiness  Level  within  Defense  Acquisition."  International  Journal  of  Defense  Acquisition  Management. 
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Key  Assumptions  and  Limitations 


•  Ordinal  data  is  given  numeric  value  in  order  to  assess  overall 
progression  or  performance. 

°  Grade  Point  Average  (GPA),  Failure  Modes  and  Effects  Analysis 
(FMEA) 

•  One  system  cannot  be  compared  to  the  SRL  of  another  system 
unless  they  are  the  same  system. 

°  You  cannot  compare  a  student  with  a  3.2  GPA  in  physics  with  a 
student  that  has  a  3.8  GPA  in  biology.  These  students  belong  to 
different  systems  of  education,  but  they  are  evaluated  with  the 
same  system  of  metrics. 

•  Analysis  is  limited  by  the  experience  of  previous  assessments  and 
experience  of  the  assessors 
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Key  Assumptions  and  Limitations 

•  Analysis  may  result  in  rank  reversals,  where  a  less  mature  SRL 
receives  a  higher  rating  than  a  more  mature  SRL. 

°  The  reason  for  this  is  that  the  rankings  are  ordinal  scale  numbers, 
and  multiplication  is  not  a  valid  operation  on  them.  The  ordinal 
rankings  only  say  that  one  ranking  is  better  or  worse  than 
another,  but  not  by  how  much. 

•  If  used  as  a  top-down  tool,  SRL  may  only  identify  major  maturity 
deficiencies  in  a  system. 

°  When  used  as  a  ,?bottom-up,?  tool  SRL  can  augment  or 
complement  other  (systems)  engineering  management  activities 
and  identify  many  more  maturity  deficiencies  resulting  in  top- 
level  symptoms. 
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What  it  can  tell  us? 


SRL 

O.IO  0.39/0.40 


0.59/0.60 


0.89/0.90  1.00 


SRL/ 


5% 


75% 


I 

20% 


15% 


Sauser,  B.,  J.  Ramirez-Marquez,  R.  Magnaye,  and  W.  Tan.  (2008).  "A  Systems  Approach  to  Expanding  the 
Technology  Readiness  Level  within  Defense  Acquisition."  International  Journal  of  Defense  Acquisition  Management. 


Managerial  Intervention 


Systems  Earned  Readiness 
Management 
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Resource  Optimization  Models  and 
System  Earned  Readiness 
Management  (SERM) 
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Generic  Development  Strategies: 

•  Optimize  development  cost  -  scoDmin  Model 

°  Magnaye,  R.,  B.  Sauser,  J.  Ramirez-Marquez,  and  W.  Tan.  (2008).  “System  Development 
Planning  Using  Readiness  Levels  in  a  Cost  of  Development  Minimization  Model.”  Systems 
Engineering. 

•  Optimize  SRL  (first-to-deploy)  —  SRLmax  Model 

n  Sauser,  B.J.  and  J.E.  Ramirez-Marquez.  (2009).  “System  Development  Planning  via  System 
Maturity  Optimization.”  IEEE  Transactions  on  Engineering  Management,  (in  press; 
available  at  http://ieeexplore.ieee.org) 

•  Optimize  system  performance  parameters 


Other  Development  Strategies: 

•  Optimize  system  value 

•  Multi-objective  optimization 
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SRL  Resource  Optimization 


/ 


Model  SRLmax  =  an  optimization  model  with  the 
objective  to  maximize  the  SRL  (a  function  of  TRL 
and  IRL)  under  constraints  associated  with 
resources. 


Year 

Target 

TRL 

IRL 

SRL 

1 
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9 

9 

9 

9 

9 

9 

9 
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9 
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8 
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8 

7 

5 

7 

2 

4 

2 

0.584 

I8 

8 

8 

6 

7 

6 

7 

7 

7 

5 

6 

2 

4 

1 

0.480 

I8 

8 

7 

6 

6 

6 

5 

6 

6 

5 

6 

2 

2 
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SRL  Resource  Optimization 


Model  SCODmin  =  an  optimization  model  whose 
objective  is  to  minimize  development  cost  (a  function  of 
TRL  and  IRL  development)  under  constraints  associated 
with  schedule  and  the  required  SRL  value. 


Model 

SRL 

TRL 

IRL 

1 

2 

3 

4 

5 

6 

1,2 

1,3 

2,3 

2,4 

3,5 

4,5 

5,6 

5 

SCODmin 

0.69 
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8 

9 

6 
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7 

SRLmax 

0.73 

8 

9 

9 

6 

9 

9 

8 

8 

8 

5 

7 
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Systems  Earned  Readiness 
Management 

Earned  value  analysis 


n  Is  a  performance  monitoring  tool 

°  Provides  a  measure  of  performance 
that  is: 

•  Realistic? 

•  Based  on  actual  data? 

°  Provides  answers  to  these  questions: 

•  What  WORK  is  scheduled  to  have  been 
completed? 

•  What  was  the  cost  estimate  for  the 
WORK  scheduled? 

•  What  WORK  has  been  accomplished? 

•  What  was  the  cost  estimated  of  the 
completed  WORK? 

•  What  have  our  actual  costs  been? 


N 

Systems  Earned 
Readiness  Analysis 

Replaces  WORK  with  MATURITY 
Using  SRLmax  and  SCODmin  makes 
it  predictive 

\ 

/ 

•  What  are  the  variances? 
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Model  Based  Systems  Engineering 


•  Utilizing  Model  Based  Systems  Engineering  (MBSE): 

°  Setup  an  environment  to  model  the  current  SRL  approach; 

°  Review  other  ‘metrics’  needed  to  be  included  and  determine 
their  relationships  to  TRL,  IRL,  and  the  System 
Architecture; 

°  Provide  a  process  for  determining  SRL  within  MBSE. 

°  Determine  a  set  of  ‘views’  or  diagrams  on  the  model 
creation  which  enables  a  way  to  communicate  (e.g.  a 
System  Maturity  Diagram) 

°  MBSE  allows  for  the  functional  decomposition  of  models 
which  would  allow  for  a  recursive  SRL  assessment  whereby 
an  SRL  at  one  level  transforms  to  a  TRL  at  another. 
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WELCOME 


PUBLICATIONS  RESEARCH  CALCULATOR  LEARN  MORE  SMA  ROUNDTABLE 


MOVING  BEYOND  TRL. 


SYSTEMS 


Development  & 
Maturity  Laboratory 


WHAT  WILL  YOU  FIND  HERE... 

The  Technology  Readiness  Level  (TRL)  scale  is  a  measure  of  maturity  of  an 
individual  technology,  with  a  view  towards  operational  use  in  a  system 
context.  A  comprehensive  set  of  concerns  becomes  relevant  when  this  metric 
is  abstracted  from  an  individual  technology  to  a  system  context,  which  may 
involve  interplay  among  multiple  technologies  that  are  integrated  through  the 
acquisition  life  cycle.  This  research  has  been  pursuing  the  development  of  a 
system-focused  approach  for  managing  system  development  and  making 
effective  and  efficient  decisions  during  the  acquisition  life  cycle.  For  this  to  be 
accomplished  our  research  has  been  evolving  in  a  series  phases: 

Phase  1  (10/06  -  09/07):  Development  of  system  maturity  indices  such  as  a 
System  Readiness  Level  (SRL)  and  Integration  Readiness  Level  (IRL)  which 
work  with  TRL  in  order  to  assess  developmental  maturity  during  the 
acquisition  life  cycle; 

Phase  2  (08/07  -  12/07):  Validation  and  calibration  of  the  systems  maturity 
indices  to  the  acquisition  life  cycle; 

Phase  3  (12/07  -  present):  Building  on  the  foundation  of  the  SRL  and 
providing  techniques  for  determining  current  and  future  readiness  of  a  system 
to  determine  its  position  in  the  acquisition  life  cycle. 


There  is  a  need  for 
"...research  in 
developing  more  formal 
metrics  and  models  for 
determining  systems 
engineering 
effectiveness  for 
enabling  continuous 
improvement ” 

-  Kelly  Miller,  Chief  Systems 
Engineer,  NSA  at  the  2005 
Conference  on  Systems 
Engineering  Research 


We  are  now  expanding  Phase  3  into  a  program  of  research  to  explore  the 
optimization  of  the  SRL  index  based  on  constrained  resources  to  provide  a 
decision  support  approach  to  enhance  managerial  capabilities  and  create  a 
systems  life  cycle  maturity  management  approach,  which  we  define  as  System 
Earned  Readiness  Management  (SERM).  Alternatively  to  Earned  Value 
Management  (EVM),  SERM  addresses  the  earned  readiness  or  maturity  of 
systems  development  as  it  eauates  to  the  acquisition  life  cycle. 


systemreadinesslevel.com 
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Washington,  DC 


FOCUS 

The  Technology  Readiness  Level  (TRL)  scale  is  a  measure  of  maturity  of  an 
individual  technology,  with  a  view  towards  operational  use  in  a  system  context  A 
comprehensive  set  of  concerns  becomes  relevant  when  this  metric  is  abstracted 
from  an  individual  technology  to  a  system  context,  which  may  involve  interplay 
among  multiple  technologies  that  are  integrated  through  a  developmental  life  cycle. 
This  research  symposium  is  focused  on  innovations  in  system  maturity  indices 
for  the  development  of  a  system-focused  approach  for  managing  system 
development  and  making  effective  and  efficient  decisions. 

PURPOSE 

The  purpose  of  this  roundtable  is  to  provide  system  designers  and  developers, 
program  and  project  managers,  and  researchers  a  platform  to  discuss  and 
disseminate  emerging  knowledge  in  systems  maturity  indices  (beyond  TRL).  The 
objective  is  to  create  a  community  of  practitioners  and  researchers  focused  on 
new  knowledge  in  system  maturity  indices  and  assessment. 

FORMAT 

For  the  first  half  of  day  one.  Presentation  Sessions  will  run  in  succession  and 
comprise  presentations  from  Stakeholders  on  the  emerging  challenges  and  potential 
solutions  in  systems  maturity  indices  and  assessment.  The  sessions  will  be  run  in  a 
panel/discussion  format  to  assure  all  attendees  are  engaged  in  the  knowledge 
exploration.  For  the  second  half  of  day  one,  breakout  groups  will  be  asked  to 
address  these  three  questions  with  respects  to  the  future  of  systems  maturity 
assessment: 

1 .  What  are  the  real  questions? 

2.  What  do  we  know? 

3.  What  do  we  need  to  know? 


» 
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SMA  Roundtable  -  Preliminary  Results 

•  What  are  the  real  questions? 

°  What  is  SRL  useful  for?  To  ensure  stakeholders  and 
users  that  the  program  is  progressing  satisfactorily. 

°  Are  existing  metrics  adequate  to  define  SRL? 

°  What  has  changed  that  makes  SRL  necessary? 
Hardware  and  software  complexity  have  increased  by 
several  orders  of  magnitude,  more  complex  interfaces, 
&  systems  are  no  longer  stovepiped. 

n  How  broadly  can  the  SRL  be  applied.  Is  it  flexible  and 
scalable  to  SoS,  software,  integration? 
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SMA  Roundtable  -  Preliminary  Results 


•  What  do  we  know? 
n  WehaveTRL 
n  Systems  are  complex 
°  Some  alternatives  exist 
°  Interfaces  are  difficult 

n  Technologies  are  changing  rapidly  (Moore’s  law) 

°  Assessment  taxonomies  vary 
°  Management  likes  single  numbers 
°  Requirements  creep  impacts  schedule  performance 
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SMA  Roundtable  -  Preliminary  Results 


•  What  do  we  need  to  know? 
n  We  need  better  data  for  acquisition  decisions 
n  We  need  better  fidelity  tools  to  aid  in  making  decisions 
(integrated  tool  set  to  handle  metrics,  portfolios, 
enterprises) 

n  We  need  to  know  how  to  validate  the  SRL  assessment 
criteria  against  actual  performance.  Are  there  any 
success  /  failure  stories  (case  studies)? 
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SMA  Roundtable  -  Preliminary  Results 

•  What  could  we  do  to  learn  that? 

n  Look  for  case  studies  for  successes/failures  and 
identify  the  data  that  warns  of  program  failure. 

n  Perform  a  tool  gap  analysis 

n  Survey  stakeholders/users  to  determine  what  criteria 
they  feel  indicates  maturity 

°  Conduct  a  pilot  program  to  validate  SRL 
°  New  issues  are  emerging  and  need  to  be  addressed 

•  System  assurance 

•  Trusted  sources 

•  Information  assurance 

•  Industrial  base  adequacy 
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